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Mr.  Chairman  and  Members  of  the  Subcommittee: 


Results  in  Brief 


I  am  pleased  to  be  here  today  to  discuss  the  application  of  best 
commercial  practices  to  Department  of  Defense  (DOD)  weapon 
acquisitions  in  general  and  to  the  Joint  Strike  Fighter  in  particular. 

After  having  done  hundreds  of  reviews  of  major  weapon  acquisitions  over 
the  last  20  years,  we  have  seen  many  of  the  same  problems  recur — cost 
increases,  schedule  delays,  and  performance  problems.  Over  the  last  4 
years,  we  have  undertaken  a  body  of  work  to  examine  weapon  acquisition 
issues  from  a  different,  more  cross-cutting  perspective.  Specifically,  we 
have  examined  the  best  product  development  practices  of  leading 
commercial  firms.  Collectively,  our  reviews  have  included  the  practices  of 
over  20  leading  commercial  firms  that  represent  a  variety  of  industries, 
including  electronics,  satellite  communications,  automotive,  medical,  and 
aircraft.  Leading  commercial  firms  are  getting  the  kind  of  results  that  DOD 
seeks:  more  sophisticated  products  developed  in  less  time  and  cost  than 
their  predecessors.  Our  work  shows  that  DOD  can  learn  valuable  lessons 
from  the  commercial  sector  to  get  better  and  more  predictable  outcomes 
from  weapon  system  development  programs.  A  listing  of  the  reports  we 
have  issued  on  best  practices  that  can  be  applied  to  weapon  acquisitions  is 
included  in  the  appendix. 

DOD  has  taken  steps  to  reflect  best  commercial  practices  in  its  acquisition 
policies.  However,  the  real  test  of  these  policies  is  in  how  they  influence 
individual  decisions,  such  as  the  upcoming  engineering  and  manufacturing 
development  decision  on  the  Joint  Strike  Fighter  program.  This  program  is 
to  produce  three  fighter  variants  to  meet  multiservice  requirements: 
conventional  flight  for  the  Air  Force,  short  take-off  and  landing  for  the 
Marine  Corps,  and  carrier  operations  for  the  Navy.  The  program  will  also 
provide  aircraft  to  the  British  royal  Navy  and  Air  Force.  As  currently 
planned,  the  program  will  cost  about  $200  billion  to  develop  and  procure 
over  3,000  aircraft  and  related  support  equipment. 

My  testimony  focuses  on  the  best  commercial  practices  for  developing 
new  products,  the  reasons  why  DOD  does  not  follow  these  practices,  and 
the  opportunity  that  Joint  Strike  Fighter  represents  to  strengthen — or 
weaken — the  effect  of  best  practices  and  acquisition  reform  on  major 
weapons. 


Leading  commercial  firms  have  adopted  a  knowledge-based  approach  to 
developing  new  products,  underwritten  by  incentives  that  encourage 
realism,  candor,  and  meeting  product  expectations.  Making  sure  that  new 
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technology  is  mature — that  is,  demonstrating  that  it  works — before  a 
product  development  starts  is  the  foundation  for  this  approach.  Leading 
commercial  firms  discipline  the  product  development  phase  by  adhering 
to  (1)  time  limits  for  completing  development  and  (2)  high  standards  for 
demonstrating  design  and  production  knowledge.  These  practices  have 
put  commercial  managers  in  an  excellent  position  to  succeed  in 
developing  better  products  in  less  time  and  producing  them  within 
estimated  costs.  To  do  otherwise  would  risk  failure  in  the  form  of 
customer  dissatisfaction. 

DOD  programs,  with  some  exceptions,  proceed  with  lower  levels  of 
knowledge  about  key  factors  of  product  development  and  allow 
technology  development  to  take  place  during  product  development.  DOD’s 
variances  from  best  practices  stem  from  strong  incentives  for  starting 
programs  too  early;  overpromising  performance  capabilities;  and 
understating  expected  costs,  schedules,  and  technical  risks.  While  these 
incentives  evolved  over  time  to  help  build  support  for  programs,  they  put 
program  managers  in  a  very  difficult  position  to  deliver  better  weapons  on 
time  and  within  budget.  Moreover,  there  is  little  risk  that  the  DOD 
customer  will  be  dismayed  by  a  program  not  being  delivered  as  promised. 
DOD  accepts  the  need  to  get  better  outcomes  from  its  weapon  system 
programs  and  accepts  best  commercial  practices  as  a  way  to  get  those 
outcomes.  In  fact,  it  is  currently  incorporating  such  practices  in  a  major 
revision  of  its  acquisition  policy.  However,  new  policies  will  not  produce 
better  program  outcomes  unless  they  influence  the  decisions  made  on 
individual  weapon  systems. 

DOD  has  designated  the  Joint  Strike  Fighter  as  a  flagship  program  for 
acquisition  reform.  Funding  requests  are  now  before  the  Congress  to 
support  the  Joint  Strike  Fighter’s  April  2001  entry  into  engineering  and 
manufacturing  development.  By  best  practices  standards,  none  of  the 
fighter’s  critical  technology  areas  identified  by  the  program  office  are 
expected  to  be  at  readiness  levels  considered  an  acceptable  risk  for  entry 
into  engineering  and  manufacturing  development.  Delaying  this  phase  of 
the  program  until  these  technologies  are  mature  would  improve  the 
chances  that  the  Joint  Strike  Fighter  will  be  fielded  as  planned.  However, 
despite  not  having  the  requisite  knowledge  for  the  eight  technologies, 

DOD  has  deemed  the  risks  manageable  and  proposes  to  proceed  with  the 
program  as  planned.  Such  a  decision  reinforces  traditional  incentives  and 
increases  the  likelihood  for  future  cost,  schedule,  and  performance 
problems.  DOD’s  plans  to  move  the  Joint  Strike  Fighter  into  engineering 
and  manufacturing  development  with  immature  technology  illustrates  a 
lack  of  commitment  to  following  best  commercial  practices  as  part  of  its 
acquisition  reform  efforts. 
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A  Best  Practices 
Model  for  Acquisition 


The  characteristics  of  best  commercial  practices  suggest  a  process  for 
developing  new  capabilities — whether  they  are  commercial  or  defense 
products — that  is  based  on  knowledge.  It  is  a  process  in  which  technology 
development  and  product  development  are  treated  differently  and 
managed  separately.  The  process  of  developing  technology  culminates  in 
discovery  and  must,  by  its  nature,  allow  room  for  unexpected  results  and 
delays.  The  process  of  developing  a  product  culminates  in  delivery,  and 
therefore,  gives  great  weight  to  design  and  production.  Discipline  is 
inherent  because  criteria  exist,  tools  are  used,  and  a  program  does  not  go 
forward  unless  a  strong  business  case  on  which  the  program  was 
originally  justified  continues  to  hold  true. 


We  have  learned  that  a  knowledge-based  process  is  essential  to  getting 
better  cost,  schedule,  and  performance  outcomes.  This  means  that 
decision  makers  must  have  virtual  certainty  about  critical  facets  of  the 
product  under  development  when  needed.  Such  knowledge  is  the  inverse 
of  risk.  Commercial  and  military  programs  do  not  all  follow  the  same 
processes  in  their  development  cycles.  However,  at  some  point,  full 
knowledge  is  attained  about  a  completed  product,  regardless  of  the 
development  approach  taken.  This  knowledge  can  be  measured  at  three 
key  junctures  that  we  refer  to  as  knowledge  points: 

•  Knowledge  point  1:  when  a  match  is  made  between  the  customer’s 
requirements  and  the  available  technology; 

•  Knowledge  point  2:  when  the  product’s  design  is  determined  to  be  capable 
of  meeting  performance  requirements;  and 

•  Knowledge  point  3:  when  the  product  is  determined  to  be  producible 
within  cost,  schedule,  and  quality  targets. 

We  have  identified  metrics  that  indicate  these  knowledge  levels  and  can 
thus  help  forecast  outcomes  as  a  development  program  progresses.  A  best 
practices  model  for  technology  development  and  product  development  is 
depicted  in  figure  1 . 
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Figure  1:  Best  Practices  for  Developing  Technology  and  Products 
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Commercial  firms  gain  more  knowledge  about  a  product’s  technology, 
design,  and  producibility  much  earlier  than  DOD  acquisition  programs  we 
reviewed.  Two  features  of  leading  commercial  products  stand  out  for 
making  a  manageable  product  development.  First,  there  is  a  clear  break 
between  technology  development  and  product  development.  The  launch  of 
a  new  product  development  in  commercial  ventures  is  a  clearly  defined 
undertaking  and  before  beginning,  firms  insist  on  having  the  technology  in 
hand  that  is  needed  to  meet  customer  requirements.  Second,  leading  firms 
limit  the  length  of  time  it  takes  to  develop  the  product.  This  limit  is  key  to 
getting  the  product  to  market  and  focuses  attention  on  the  design  and 
production  knowledge  points.  It  also  provides  discipline  to  the  technology 
development  process  to  ensure  product  development  will  not  be  launched 
until  the  technology  match  is  made. 

The  leading  commercial  firms  we  have  visited  consciously  limited  their 
product  developments  from  18  months  to  just  over  4  years.  They 
understand  that  this  keeps  product  development  within  a  time  frame  that 
can  keep  people  focused  on  delivering  a  product.  In  fact,  one  commercial 
executive  observed  that  it  is  unreasonable  to  expect  people  to  focus  on  a 
goal  like  production  start  that  is  more  than  4  years  away.  The  limited  time 
frames  provide  strong  incentives  for  a  commercial  manager  to  keep 
immature  technologies  out  of  the  product  design.  In  fact,  these  time 
frames  give  product  managers  clout  in  fostering  cost  and  performance 
trade-offs  before  the  program  begins,  ultimately  limiting  a  product’s 
requirements  to  what  can  be  achieved  with  demonstrated  technologies 


Page  4 


GAO/T-NSIAD-OO-173 


within  the  specified  time  period.  To  live  within  these  time  constraints  and 
keep  innovation  alive,  commercial  firms  have  adopted  an  evolutionary 
approach;  they  save  requirements  that  cannot  be  met  with  proven 
technologies  for  the  next  iteration  of  the  product.  Commercial  firms  also 
found  that  limiting  product  development  time  frames 

•  makes  it  easier  to  hold  people  accountable  for  meeting  promised  cost, 
schedule,  and  performance  targets; 

•  enables  a  production-oriented  focus  throughout  product  development, 
providing  incentives  for  identifying  risks  early; 

•  makes  product  development  costs  and  schedule  more  predictable;  and 

•  allows  firms  to  get  into  production  and,  therefore,  to  the  point  of  sale 
quicker. 

Once  a  product  development  is  under  way,  the  firms  demand — and 
receive — specific  knowledge  about  the  design  and  producibility  of  the 
product  before  production  begins.  The  process  of  discovery — the 
accumulation  of  knowledge  and  elimination  of  unknowns — is  completed 
well  ahead  of  production.  There  is  a  synergy  in  this  process,  as  the 
attainment  of  each  successive  knowledge  point  builds  on  the  preceding 
one. 

In  contrast,  DOD  programs  are  started  earlier  and  allow  technology 
development  to  continue  into  product  development  and  even  into 
production.  Consequently,  the  programs  proceed  with  much  less 
knowledge  available — and  thus  more  risk — about  required  technologies, 
design  capability,  and  producibility.  This  approach  to  technology  and 
product  development  is  shown  in  figure  2. 
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Figure  2:  Levels  of  Knowledge  Attained  in  DOD  Weapon  System 
Developments 
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Proceeding  with  lower  levels  of  knowledge  available  means  that  during 
product  development,  maturity  of  technology,  the  design,  and  production 
methods  must  all  be  pursued  at  the  same  time.  The  rippling  effect  of 
discovering  and  overcoming  problems  in  product  development  explains 
much  of  the  turbulence  in  DOD  program  outcomes.  Metrics,  such  as 
technology  readiness  levels  and  percent  of  engineering  drawings 
complete,  can  be  used  to  predict  these  consequences.  Product 
development  times,  long  to  begin  with,  stretch  even  further  in  reaction  to 
problems.  We  calculate  that  they  can  take  3  to  10  times  as  long  as 
commercial  products. 


Knowledge  P oint  1 :  Technology  development  has  the  ultimate  objective  of  bringing  a 

Requirements  and  technology  up  to  the  point  that  it  can  be  readily  integrated  into  a  new 

Technology  Are  Matched  product  and  counted  on  to  meet  requirements.  We  have  found  that  getting 

match  between  customer  requirements  and  mature  technology  to  be 
the  biggest  contributor  to  a  successful  product  development.1  As  a 


1  For  more  information,  see  Best  Practices:  Better  Management  of  Technology  Development  Can 
Improve  Weapon  System  Outcomes  (GAO/NSIAD-99-1G2,  July  30.  1999). 
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technology  is  developed,  it  moves  from  a  concept  to  a  feasible  invention  to 
a  component  that  must  fit  onto  a  product  and  function  as  expected.  In 
between,  there  are  increasing  levels  of  demonstration  that  can  be 
measured.  In  our  review  of  best  practices  for  including  new  technology  in 
products,  we  applied  a  scale  of  technology  readiness  levels — from  one  to 
nine— pioneered  by  the  National  Aeronautics  and  Space  Administration 
and  adapted  by  the  Air  Force  Research  Laboratories. 

Without  going  into  the  details  of  each  level,  a  level  four  equates  to  a 
laboratory  demonstration  of  a  technology  that  is  still  not  in  a  usable  form. 
Imagine  an  advanced  radio  technology  that  can  be  demonstrated  with 
components  that  take  up  a  table  top.  When  initial  hand-built  versions  of 
all  of  the  radio’s  basic  elements  are  hand-wired  and  tested  together  in  a 
laboratory,  the  radio  reaches  a  readiness  level  of  five.  A  technology 
readiness  level  of  seven  is  the  demonstration  of  a  technology  that 
approximates  its  final  form  and  occurs  in  an  environment  outside  the 
laboratory.  The  same  radio  at  level  seven  would  be  installed  and 
demonstrated  in  a  platform,  such  as  a  fighter  aircraft. 

The  lower  the  level  of  the  technology  at  the  time  it  is  included  in  a  product 
development  program,  the  higher  the  risk  that  it  will  cause  problems  in  the 
product  development.  According  to  the  Air  Force  Research  Laboratory, 
level  seven  enables  a  technology  to  be  included  on  a  product  development 
with  acceptable  risk.  When  we  asked  leading  commercial  firms  to  apply 
these  standards  to  their  own  methods  for  assessing  technology  maturity, 
we  found  that  most  insisted  on  even  higher  levels  of  readiness  before  they 
allowed  a  new  technology  into  product  development.  When  we  examined 
weapon  systems  that  experienced  cost  and  schedule  problems,  we  found 
that  they  started  with  key  technologies  at  levels  three  and  four.  By  the 
time  the  programs  reached  a  point  DOD  considers  analogous  to  beginning 
product  development,  key  technologies  were  still  at  level  five  or  lower. 
Conversely,  DOD  programs  for  which  key  technologies  were  at 
significantly  higher  levels  of  maturity  at  the  start,  had  not  experienced 
such  problems. 


Knowledge  Point  2:  The 
Design  Will  Perform  as 
Required 


By  the  halfway  point  in  product  development,  leading  commercial  firms 
achieve  near  certainty  that  their  product  designs  would  meet  customer 
requirements  and  have  gone  a  long  way  toward  ensuring  that  the  products 
can  be  produced.2  Both  DOD  and  the  commercial  firms  hold  a  critical 


2  For  more  information,  see  Best  Practices:  Successful  Application  to  Weapon  Acquisitions  Require 
Changes  in  DOD’s  Environment  (GAO/NSIAD-98-56,  Feb.  24,  1998). 
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design  review  to  review  engineering  drawings,  confirm  the  design  is 
mature,  and  “freeze”  the  design  to  minimize  changes  in  the  future.  The 
completion  of  engineering  drawings  and  their  release  to  manufacturing 
organizations  signify  that  program  managers  are  confident  in  their 
knowledge  that  the  design  performs  acceptably  and  is  mature.  The 
drawings  are  critical  to  documenting  this  knowledge  because  they  are 
precision  schematics  of  the  entire  product  and  all  of  its  component  parts. 
They  also  reflect  the  results  of  testing  and  simulation  and  describe  the 
materials  and  manufacturing  processes  to  be  used  to  make  each 
component. 

Both  DOD  and  commercial  companies  consider  the  design  to  be 
essentially  complete  when  about  90  percent  of  the  engineering  drawings 
are  completed.  Officials  from  commercial  companies  such  as  Boeing  and 
Hughes  told  us  that  they  typically  had  over  90  percent  of  these  drawings 
available  for  the  critical  design  review.  The  DOD  programs  we  reviewed 
had  less  than  60  percent— one  had  less  than  one-third— of  the  drawings 
done  at  the  time  their  critical  design  reviews  were  held.  Thus,  these 
programs  had  significantly  less  knowledge  about  their  designs.  The 
programs  did  not  get  or  were  not  expected  to  get  to  the  90-percent  level  of 
completion  on  the  drawings  until  late  in  development  or  in  production. 
Nonetheless,  at  the  time  of  the  critical  design  reviews,  the  risks  of 
proceeding  with  the  rest  of  development  on  these  programs  as  planned 
were  deemed  acceptable.  The  programs  however  encountered  significant 
design  problems  in  testing  that  occurred  after  the  critical  design  review. 


Knowledge  Point  3: 
Production  Units  Will  Meet 
Cost,  Quality,  and  Schedule 
Objectives 


Leading  commercial  firms  reach  the  point  at  which  they  know  that 
manufacturing  processes  will  produce  a  new  product  conforming  to  cost, 
quality,  and  schedule  targets  by  the  end  of  product  development — before 
they  begin  fabricating  production  articles.  Reaching  this  point  means  more 
than  knowing  the  product  could  be  manufactured;  it  means  that  all  key 
processes  are  under  control,  such  that  the  quality,  volume,  and  cost  of 
their  output  are  proven  acceptable.  Commercial  firms  relied  on  good 
supplier  relationships,  known  manufacturing  processes,  and  statistical 
process  control  to  achieve  this  knowledge  early  and,  in  fact,  had  all  their 
key  processes  under  statistical  process  control  when  production  begins. 
All  of  the  companies  we  visited  agreed  that  a  high  level  of  knowledge 
about  technology  and  design  early  in  the  process  makes  the  control  of 
processes  possible. 


DOD  programs  did  not  have  nearly  this  level  of  knowledge  at  production. 
One  weapon  system  program  that  had  been  in  production  for  nearly  9 
years  at  the  time  of  our  1998  review  still  had  less  than  13  percent  of  its  key 
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manufacturing  processes  in  control.  Another  program  had  40  percent  of  its 
key  manufacturing  processes  in  control  2  years  before  production  was 
scheduled  to  begin  but  was  not  scheduled  to  have  all  key  processes  in 
control  until  4  years  into  production.  Both  programs  experienced  basic 
producibility  problems  that  were  not  discovered  until  late  in  development 
or  early  in  production.  These  risks  went  unrecognized  even  though  DOD 
had  established  criteria  for  determining  whether  risks  were  acceptable 
and  whether  enough  knowledge  had  been  gained  to  enter  the  next 
development  phase. 


Changes  in  DOD 
Environment  Needed 
to  Adopt  Best 
Practices 


The  most  important  factors  in  the  adoption  of  best  practices  are  the 
incentives  perceived  by  managers  of  technology  and  product 
developments.  The  differences  in  the  practices  employed  by  the  leading 
commercial  firms  and  DOD  reflect  the  different  demands  imposed  on 
programs  in  their  environments.  The  way  success  and  failure  are  defined 
for  commercial  and  defense  product  developments  differs  considerably 
and  results  in  different  incentives,  evoking  different  behaviors  from  the 
people  managing  the  programs.  Specific  practices  take  root  and  are 
sustained  because  they  help  a  program  succeed  in  its  environment. 


Incentives  in  the  Leading  commercial  firms  begin  product  development  only  when  a  solid 

Commercial  Environment  business  case  can  be  made.  The  business  case  centers  on  the  ability  to 

produce  a  product  that  the  customer  will  buy  and  that  will  provide  an 
acceptable  return  on  investment.  The  point  of  sale  occurs  in  production 
after  development  is  complete;  program  success  is  determined  when  the 
customer  buys  the  finished  product.  If  the  firm  has  not  made  a  sound 
business  case,  or  has  been  unable  to  deliver  on  one  or  more  of  the 
business  case  factors,  it  faces  a  very  real  prospect  of  failure — the 
customer  may  walk  away.  Also,  if  one  product  development  takes  more 
time  and  money  to  complete  than  expected,  it  denies  the  firm 
opportunities  to  invest  those  resources  in  other  products.  Because  the 
match  between  technologies  and  product  requirements  is  made  before  the 
product  development  is  launched,  the  cost  and  schedule  consequences 
associated  with  discovery  are  minimized. 

Production  is  a  dominant  concern  throughout  the  product  development 
process  and  forces  discipline  and  trade-offs  in  the  design  process.  This 
environment  encourages  realistic  assessments  of  risks  and  costs;  doing 
otherwise  would  threaten  the  business  case  and  invite  failure.  For  the 
same  reasons,  the  environment  places  a  high  value  on  knowledge  for 
making  decisions.  Program  managers  have  good  reasons  to  identify  risks 
early,  be  intolerant  of  unknowns,  and  not  rely  on  testing  late  in  the  process 
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Incentives  in  the  DOD 
Environment 


as  the  main  vehicle  for  discovering  the  performance  characteristics  of  the 
product.  By  protecting  the  business  case  as  the  key  to  success,  program 
managers  in  leading  commercial  firms  are  conservative  in  their  estimates 
and  aggressive  in  risk  reduction.  Ultimately,  preserving  the  business  case 
strengthens  the  ability  of  managers  to  say  “no”  to  pressures  to  accept  high 
risks  or  unknowns.  Practices  such  as  maturing  technologies  to  high 
readiness  levels  before  inclusion  in  a  program,  having  90  percent  of 
engineering  drawings  done  by  the  critical  design  review,  and  achieving 
statistical  process  control  before  production  are  adopted  because  they 
help  ensure  success. 


The  basic  management  goal  for  a  weapon  system  program  in  DOD  is 
similar:  to  develop  and  deliver  a  product  that  meets  the  customer’s  needs. 
However,  the  pressures  of  successfully  competing  for  the  funds  to  start 
and  sustain  a  DOD  program  provide  different  incentives.  Compared  with 
commercial  programs,  the  DOD  environment  encourages  launching 
product  developments  that  embody  more  technical  unknowns  and  less 
knowledge  about  the  performance  and  production  risks  they  entail.  A  new 
weapon  system  can  be  more  readily  defended  if  it  possesses  performance 
features  that  significantly  distinguish  it  from  other  systems.  Consequently, 
aspiring  DOD  programs  have  incentives  to  include  performance  features 
and  design  characteristics  that  rely  on  immature  technologies.  These 
unknowns  place  a  much  greater  focus  on  maturing  technology  during 
product  development  than  we  found  on  commercial  programs. 

Even  though  less  information  about  a  new  product  development  is 
available  at  the  time  DOD  programs  are  launched,  the  competition  for 
funding  forces  detailed  projections  to  be  made  from  that  information.  A 
product  development  cannot  be  launched  unless  the  program’s 
development  and  production  cost,  as  well  as  timing,  falls  within  available 
funding.  Because  DOD  relies  largely  on  forecasts  of  cost,  schedule,  and 
performance  that  are  comparatively  soft  at  the  time,  success  in  competing 
for  funding  encourages  managers  to  squeeze  cost  and  schedule  estimates 
into  profiles  of  available  funding.  Additional  requirements,  such  as  high 
reliability  and  maintainability,  serve  to  make  the  fit  even  tighter.  As 
competition  for  funding  will  continue  throughout  the  program’s 
development,  success  is  measured  in  terms  of  ability  to  secure  the  next 
installment. 

The  risks  associated  with  developing  new  technologies  together  with  the 
product — within  tight  estimates — are  deemed  acceptable.  Production 
realities,  critical  to  matching  technological  capabilities  with  customer 
requirements  on  commercial  programs,  are  too  far  away  from  the  DOD 
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launch  decision  to  have  the  same  curbing  effect  on  technology  decisions. 
The  environment  for  managing  weapon  system  programs  is  particularly 
difficult  for  managing  technology  development.  The  ups  and  downs  and 
resource  changes  associated  with  the  technology  discovery  process  do  not 
mesh  well  with  a  program’s  need  to  meet  cost,  schedule,  and  performance 
goals.  Problems  with  developing  technologies,  which  are  to  be  expected, 
can  actually  threaten  the  support  for  a  program  if  they  become  known. 

These  pressures  and  incentives  explain  why  the  behavior  of  weapon 
system  managers  differs  from  managers  of  commercial  product 
developments.  Problems  or  indications  that  the  estimates  are  being 
breached  do  not  help  sustain  funding  support  for  the  program  in  later 
years,  and  thus  their  admission  is  implicitly  discouraged.  An  optimistic 
cost  estimate  makes  it  easier  to  launch  a  product  development  and  sustain 
annual  approval;  admission  that  costs  are  likely  to  be  higher  could  invite 
failure.  Rewards  for  discovering  and  recognizing  potential  problems  early 
in  a  DOD  product  development  are  few.  Less  available  knowledge  makes  it 
harder  for  program  managers  to  say  “no.”  In  contrast  with  leading 
commercial  firms,  not  having  attained  knowledge — on  the  full 
performance  of  a  key  technology  or  the  true  risks  facing  manufacture,  for 
example, — can  be  perceived  as  better  than  knowing  that  problems  exist. 
For  these  reasons,  the  practices  associated  with  managing  to  knowledge 
standards — such  as  for  technology,  design,  and  production  maturity — are 
not  readily  adopted  in  DOD. 

These  observations  about  the  differences  between  the  commercial  and 
DOD  environments  should  not  be  interpreted  to  mean  that  commercial 
managers  are  somehow  more  skilled  or  knowledgeable  than  their  DOD 
counterparts.  DOD  program  managers  act  in  response  to  the  pressures 
they  face.  All  of  the  numerous  participants  in  the  acquisition  process  play 
a  part  in  creating  these  pressures.  Commercial  program  managers  are  put 
in  a  better  position  to  succeed;  they  have  to  worry  only  about  product 
design  and  production  within  the  cost,  schedule,  and  performance 
demands  of  the  business  case. 


Charting  a  Course  for 
Adopting  Best  Practices  to 
Get  Better  Outcomes 


Commercial  practices  for  gaining  knowledge  and  assessing  risks  can  help 
produce  better  outcomes  on  weapon  systems.  Collectively,  better 
individual  outcomes  will  help  DOD  to  attain  modernization  goals  and 
improve  funding  stability  for  programs.  For  such  practices  to  work, 
however,  the  knowledge  they  produce  must  help  a  DOD  program  succeed 
in  its  environment.  Thus,  the  DOD  environment  must  become  conducive  to 
such  practices.  At  least  two  factors  are  critical  to  fostering  such  an 
environment.  First,  managers  must  be  relieved  of  the  need  to  overpromise 
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on  performance  and  resource  estimates  at  the  program  launch  decision. 
Separating  technology  development  so  that  it  does  not  have  to  be  managed 
within  the  confines  of  a  weapon  system  program  would  go  a  long  way  to 
relieving  this  pressure.  Clearly,  DOD  has  to  develop  technology, 
particularly  the  technology  that  is  unique  to  military  applications. 

However,  by  separating  technology  development  from  weapon  programs, 
DOD  could  insist  on  higher  standards  for  knowledge  on  its  programs  and 
get  better  results  when  those  programs  transition  to  production. 

Second,  once  a  program  is  under  way,  the  participants  in  the  acquisition 
process  must  make  it  acceptable  for  managers  to  identify  unknowns  as 
high  risks  so  that  they  can  be  aggressively  worked  on  earlier  in 
development.  If  the  Congress  and  DOD  weighed  program  launch  decisions 
and  subsequent  progress  on  weapon  systems  by  applying  a  common  set  of 
knowledge  standards,  like  those  gleaned  from  leading  commercial  firms, 
they  could  create  a  better  business  case  for  starting  a  weapon  system 
program.  By  developing  technology  separately  to  high  readiness  levels 
before  including  it  in  a  program  and  by  adhering  to  standards  such  as 
knowledge  points  in  product  development,  DOD  program  managers  can  be 
put  in  a  better  position  to  succeed  in  the  timely  design  and  production  of 
weapon  systems.  The  shorter  cycle  times  associated  with  these  practices 
could  make  it  possible  to  better  align  the  tenures  of  program  managers 
with  the  product  development  phase,  making  them  more  accountable  for 
program  outcomes. 

The  Office  of  the  Under  Secretary  of  Defense  (Acquisition,  Technology, 
and  Logistics)  supports  shorter  weapon  system  development  times  and 
more  aggressive  pursuit  of  technology  outside  of  programs.  It  also 
supports  the  use  of  best  commercial  practices,  such  as  taking  an 
evolutionary  approach  to  developing  new  products.  DOD  is  capturing 
these  and  other  practices  in  a  substantial  revision  of  the  regulations  that 
guide  the  management  of  weapon  system  programs.  These  regulations  are 
currently  in  draft  form. 

The  real  test  of  the  participants’  resolve  to  get  better  outcomes  by  applying 
best  practices  will  be  the  decisions  made  on  individual  weapon  systems, 
such  as  for  launch  and  funding.  These  decisions  define  what  success 
means  in  DOD  and  what  practices  contribute  to  success.  Decisions  made 
by  DOD  or  the  Congress  to  advance  or  fund  programs  that  do  not  have 
enough  knowledge  to  meet  agreed-upon  standards  signal  to  managers  that 
not  having  the  necessary  level  of  knowledge  is  acceptable.  On  the  other 
hand,  participants  should  support  decisions  to  not  start  new  programs  that 
need  technology  advances  to  meet  unforgiving  requirements  or  to 
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recognize  early  that  a  change  in  a  program  is  necessary  to  attain  desired 
knowledge  levels. 


The  Decision  to  Move 
the  Joint  Strike 
Fighter  Into 
Engineering  and 
Manufacturing 
Development:  A  Case 
in  Point 


The  Joint  Strike  Fighter  program,  and  its  implications  for  best  practices 
and  true  acquisition  reform,  is  a  good  test  of  whether  the  desire  for  better 
outcomes  can  outweigh  traditional  pressures  to  get  a  program  approved. 
In  our  recent  review  of  the  Joint  Strike  Fighter  program,  we  employed 
knowledge  standards  consistent  with  best  practices  and  DOD  acquisition 
reforms. 3  DOD  has  designated  the  Joint  Strike  Fighter  program  as  a 
flagship  program  for  acquisition  reform.  The  Joint  Strike  Fighter  program, 
the  most  expensive  aircraft  program  in  DOD,  is  at  a  critical  juncture  in  its 
acquisition  cycle.  It  is  approaching  the  point  where  DOD  must  decide 
whether  to  commit  to  the  engineering  and  manufacturing  development 
phase — analogous  to  a  commercial  product  launch  decision.  During 
engineering  and  manufacturing  development,  the  Joint  Strike  Fighter  will 
be  fully  developed,  engineered,  designed,  fabricated,  tested,  and  evaluated 
to  demonstrate  that  the  production  aircraft  will  meet  stated  requirements. 
This  phase  is  estimated  to  cost  $20  billion,  require  annual  funding  levels  as 
high  as  $4  billion,  and  last  about  8  years. 


The  best  practice  for  such  a  decision  is  to  have  a  match  between  mature 
technologies  and  weapon  requirements.  It  represents  the  first  knowledge 
point.  The  Joint  Strike  Fighter  does  not  meet  this  standard;  several 
technologies  that  are  critical  to  meeting  requirements  will  not  be 
sufficiently  mature.  Consequently,  the  Joint  Strike  Fighter  will  not  enter 
the  engineering  and  manufacturing  development  phase  with  low  technical 
risk.  However,  DOD  would  like  to  go  forward  with  the  program  anyway. 
Doing  so  would  have  two  major  consequences.  First,  it  would  put  the 
program  on  a  path  that  has  yielded  cost  growth  and  schedule  slippage  on 
many  previous  programs.  Second,  as  Joint  Strike  Fighter  is  the  largest 
acquisition  in  the  foreseeable  future,  it  will  send  signals  to  other  programs 
that  best  practices  and  acquisition  reform  need  not  be  heeded  when  it 
comes  to  major  weapon  systems. 


Joint  Strike  Fighter 
Requirements  Depend  on 
Immature  Technologies 


While  we  have  been  encouraged  by  the  design  of  the  Joint  Strike  Fighter 
acquisition  strategy,  we  have  become  concerned  about  its  implementation. 
Our  biggest  concern  is  that  critical  technologies  for  meeting  affordability 
and  performance  requirements  are  projected  to  be  at  low  levels  of 


3  Joint  Strike  Fighter  Acquisition:  Development  Schedule  Should  Be  Changed  to  Reduce  Risks 
(G  AO/N  SI  AD-00-74 ,  May  9,  2000). 
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maturity  if  the  engineering  and  manufacturing  development  contract 
begins  as  scheduled  in  April  2001.  In  other  words,  the  program  may  not 
achieve  a  fundamental  element  of  a  knowledge-based  process — the 
separation  of  technology  development  from  product  development — as  it 
begins  full-fledged  product  development  activities.  This  means  that  the 
Joint  Strike  Fighter  program  manager  will  be  responsible  for  developing 
technologies  while  concurrently  designing,  building,  and  testing  the 
prototype  Joint  Strike  Fighter  aircraft.  On  past  programs,  such 
concurrency  has  caused  significant  cost  growth  and  schedule  delays.  This 
is  the  first,  and  perhaps  most  important,  knowledge  point  exhibited  in  best 
practices,  because  lack  of  knowledge  about  technologies  means  the 
program  manager  cannot  fully  focus  on  design  and  manufacturing  issues. 
This  additional  risk  makes  product  development  cycle  time  and  cost  less 
predictable.  In  addition,  once  in  a  product  development  environment, 
external  pressures  to  keep  a  program  moving  become  dominant,  such  as 
preserving  cost  and  schedule  estimates  to  secure  budget  approval. 

In  our  recently  issued  report,  we  evaluated  the  maturity  of  key 
technologies  on  the  Joint  Strike  Fighter  program.  At  our  request,  the 
program  office  identified  eight  technology  areas  that  are  considered 
critical  to  meeting  the  fighter’s  cost,  schedule,  and  performance 
objectives.  In  conjunction  with  the  program  office  and  the  two  competing 
contractors,  we  determined  the  readiness  levels  of  these  technologies 
needed  to  meet  Joint  Strike  Fighter  performance  requirements  at  three 
points  in  time:  when  the  Joint  Strike  Fighter  program  was  started  in  1996, 
when  we  conducted  our  review  in  December  1999,  and  when  the  program 
is  scheduled  to  enter  engineering  and  manufacturing  development.  Those 
assessments  showed  that  when  the  Joint  Strike  Fighter  program  was 
started,  most  of  the  critical  technologies  were  well  below  the  readiness 
levels  considered  acceptable  risk  to  begin  a  program.  The  technology 
readiness  levels  of  the  eight  critical  Joint  Strike  Fighter  technology  areas 
are  shown  in  figure  3. 
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Figure  3:  Joint  Strike  Fighter  Critical  Technology  Readiness  Levels  at 
Program  Start  and  as  Projected  for  Entry  into  Engineering  and 
Manufacturing  Development 
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As  figure  3  shows,  all  of  the  critical  technology  areas  are  expected  to  be  at 
readiness  levels  lower  than  the  level  seven  considered  acceptable  risk  for 
entry  into  engineering  and  manufacturing  development.  Six  of  the 
technologies  will  still  be  below  the  readiness  level  that  is  considered 
acceptable  risk  for  program  start,  which  occurred  over  3  years  ago  for  the 
Joint  Strike  Fighter  program.  Many  of  these  will  only  be  demonstrated  in 
laboratories  or  in  ground  tests  when  the  engineering  and  manufacturing 
development  phase  starts.  They  have  a  considerable  amount  of 
development  remaining  before  they  are  considered  mature.  Moreover,  as  a 
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result  of  cost  growth  and  schedule  concerns,  in  May  1999,  DOD  delayed 
some  technology  demonstrations  until  after  the  engineering  and 
manufacturing  development  phase  begins. 

Should  any  of  these  technologies  be  delayed  or,  worse,  not  available  for 
incorporation  into  the  final  Joint  Strike  Fighter  design,  the  impact  on  the 
program  would  be  dramatic.  For  example,  if  one  of  the  above  critical 
technologies  needed  to  be  replaced  with  its  planned  backup,  DOD  could 
expect  an  increase  of  several  billion  dollars  in  production  and  operation 
and  support  costs.4  The  backup  technology  would  also  significantly 
increase  aircraft  weight,  which  can  negatively  impact  aircraft 
performance.  This  technology  is  expected  to  be  at  unacceptable  readiness 
levels  at  the  beginning  of  the  engineering  and  manufacturing  development 
phase,  which  indicates  that  substantial  technology  development  must  still 
occur  during  this  phase. 

As  noted  above,  at  the  policy  level,  DOD  officials  have  agreed  that 
technology  development  should  be  kept  separate  from  product 
development  and  that  technology  readiness  levels  are  a  valid  way  to  assess 
technology  maturity.  However,  in  its  response  to  our  report  on  the  Joint 
Strike  Fighter — an  individual  program  decision — DOD  balked  at  the  use  of 
technology  readiness  levels  and  their  implications  for  keeping  technology 
development  out  of  the  Joint  Strike  Fighter’s  engineering  and 
manufacturing  development  phase.  One  of  the  reasons  DOD  cited  for  its 
unwillingness  to  accept  the  technology  readiness  levels  assessed  for  the 
eight  Joint  Strike  Fighter  technologies  was  that  the  levels  were  based  on 
integration  in  the  Joint  Strike  Fighter  aircraft — too  high  a  standard.  On  the 
contrary,  the  technology  readiness  levels  assessed  by  the  program  office 
and  the  contractors  were  based  on  a  clear  understanding  that  a  level  seven 
could  be  reached  by  demonstrating  a  technology  in  a  relevant 
environment.  It  was  further  made  clear  that  a  relevant  environment  would 
include  demonstrating  a  technology  in  an  existing  aircraft  like  an  F-16,  not 
a  Joint  Strike  Fighter. 

A  second  reason  DOD  disagreed  with  the  readiness  levels  assessed  for  the 
eight  technologies  was  that  its  own  risk  mitigation  plans  and  judgment 
were  more  meaningful  and  that  they  showed  the  technology  risk  to  be 
acceptable.  Risk  mitigation  plans  and  judgment  are  necessary  to  managing 
any  major  development  effort  like  the  Joint  Strike  Fighter.  However, 


4  Specific  details  cannot  be  provided  due  to  the  competitive  nature  of  the  Joint  Strike  Fighter  program. 
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without  an  underpinning  such  as  technology  readiness  levels  that  allows 
transparency  into  program  decisions,  these  methods  allow  significant 
technical  unknowns  to  be  judged  acceptable  risks  because  a  plan  exists 
for  resolving  the  unknowns  in  the  future.  Experience  on  previous 
programs  has  shown  that  such  methods  have  rarely  assessed  technical 
unknowns  as  a  high  or  unacceptable  risk;  consequently,  they  failed  to 
guide  programs  to  meet  promised  outcomes.  Technology  readiness  levels 
are  based  on  actual  demonstrations  of  how  well  technologies  actually 
perform.  Their  strength  lies  in  the  fact  that  they  characterize  knowledge 
that  exists  rather  than  plans  to  gain  knowledge  in  the  future;  they  are  thus 
less  susceptible  to  optimism.  A  clear  picture  of  knowledge — or  its 
absence — may  be  more  likely  to  prompt  action  than  a  favorable  risk 
assessment. 


An  8-Year  Phase  That 
Spans  Technology 
Development  to 
Production  Represents  a 
Herculean  Management 
Undertaking 


The  Joint  Strike  Fighter  program  began  in  1996  and  will  not  deliver  its  first 
operational  aircraft  until  2008.  With  the  4  years  spent  in  concept 
demonstration  and  the  8  years  to  be  spent  in  engineering  and 
manufacturing  development,  the  result  is  a  12-year  development  cycle. 
This  is  much  longer  than  the  development  cycles  of  leading  commercial 
firms  and  double  the  goal  set  by  the  Office  of  the  Secretary  of  Defense. 
Much  of  this  long  development  cycle  is  the  result  of  the  need  to  mature 
technologies  that  have  not  yet  been  demonstrated  as  ready  to  meet  key 
cost  and  performance  requirements.  DOD  has  highlighted  the  Joint  Strike 
Fighter  program  as  one  that  will  make  significant  cost/performance  trade¬ 
offs  in  order  to  develop  an  affordable  aircraft.  However,  DOD’s  desire  to 
have  a  low  cost  aircraft  that  must  also  meet  demanding  requirements  has 
limited  the  range  of  technological  solutions  and  has  necessitated  the 
pursuit  for  new  technologies. 

Traditionally,  a  weapon’s  final  performance  requirements  are  developed 
early  in  a  program,  or  in  many  cases  before  the  program  begins.  In  the 
case  of  Joint  Strike  Fighter  program,  requirements  were  finalized  much 
later  in  the  acquisition  cycle.  Program  officials  stated  that  this  provided 
the  program  flexibility  to  conduct  cost  and  performance  trade-offs  before 
requirement  and  design  decisions  became  final.  While  this  approach  is 
consistent  with  best  practices,  it  has  not  adequately  taken  into  account  the 
readiness  of  the  critical  technologies.  Many  of  the  trade-offs  that  were 
made  involved  decisions  to  bring  technologies  that  were  not  yet 
demonstrated  into  the  engineering  and  manufacturing  phase  of  the 
program.  Thus,  the  program  does  not  have  a  baseline  design  based  on 
demonstrated  technologies  that  could  be  developed  in  cycle  times 
commensurate  with  best  practices. 
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As  the  Joint  Strike  Fighter  program  approaches  the  engineering  and 
manufacturing  development  decision  point,  it  is  not  in  a  good  position  to 
succeed,  if  success  means  delivering  the  aircraft  on  time  and  within 
budget.  The  design  requires  significant  technological  invention  in  order  to 
satisfy  all  the  user’s  requirements.  According  to  Joint  Strike  Fighter 
officials,  an  objective  of  the  engineering  and  manufacturing  development 
phase  is  bringing  the  technologies  up  to  maturity  levels  that  will  allow 
them  to  be  incorporated  onto  a  Joint  Strike  Fighter.  Therefore,  at  a  time 
when  the  program  should  be  focused  on  designing  and  building  the 
aircraft,  the  Joint  Strike  Fighter  program  will  have  to  put  significant  effort 
and  resources  into  demonstrating  that  key  technologies  are  ready  for 
inclusion  onto  the  product.  As  a  result,  the  program  has  planned  almost  8 
years  for  its  engineering  and  manufacturing  development  phase.  The 
length  and  scope  of  the  effort  operate  against  the  ability  to  estimate  cost 
and  completion  schedules. 

Commercial  firms  have  established  practices  to  limit  product  development 
cycle  times,  thereby  increasing  the  possibility  that  program  managers  will 
remain  on  programs  until  they  are  complete.  Holding  one  program 
manager  accountable  for  the  content  of  a  product  at  the  time  the  launch 
decision  is  made  encourages  that  person  to  raise  issues  and  problems 
early  and  not  overpromise  the  capabilities  of  a  new  product  by  relying  on 
immature  technologies.  This  puts  the  manager  in  a  good  position  to  deliver 
a  high  quality  product,  on  time,  and  on  budget. 

Since  the  next  phase  of  the  Joint  Strike  Fighter  program  is  estimated  to 
last  about  8  years,  program  managers  currently  involved  in  key  decisions 
about  the  development  plan  will  likely  not  be  responsible  for  its 
implementation.  It  has  already  had  three  program  managers  since  its 
beginning  about  3  Vi  years  ago.  As  a  result,  conditions  to  be  accepted  at 
engineering  and  manufacturing  development,  such  as  the  acceptance  of 
low  technology  readiness  levels,  will  more  than  likely  become  the 
responsibility  of  another  program  manager. 


Conclusion 


At  this  point  in  its  development,  there  are  a  number  of  ways  for  DOD  to 
make  the  Joint  Strike  Fighter  program’s  environment  more  conducive  to 
better  cost  and  schedule  outcomes.  We  believe  that  separating  technology 
development  from  product  development  can  still  create  conditions  for  a 
successful  Joint  Strike  Fighter  program.  To  proceed  as  planned — entering 
a  phase  of  the  program  with  immature  technologies  that  should  be  focused 
on  design  and  production — is  to  risk  continued  delays  and  cost  growth. 
Instead,  the  program  has  an  opportunity  to  mature  technologies  in  a  more 
risk-tolerant  environment  by  making  the  right  decisions  now. 
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In  our  report,  we  recommend  that  the  Joint  Strike  Fighter  program 
continue  in  its  current  program  definition  and  risk  reduction  phase, 
delaying  the  decision  to  move  into  engineering  and  manufacturing  until 
technologies  are  demonstrated  to  acceptable  levels.  Taking  the  additional 
time  to  mature  the  technologies  will  then  allow  the  program  manager  to 
focus  on  design  and  manufacturing  risks  during  engineering  and 
manufacturing  development.  It  also  increases  the  possibility  of  completing 
product  development  in  a  more  timely  and  predictable  manner.  Such  a 
delay  does  not  necessarily  lengthen  the  total  product  development  cycle. 
In  fact,  the  knowledge  gained  from  time  spent  developing  technologies  in 
the  beginning  can  often  shorten  the  time  it  takes  to  get  the  product  to 
market. 

Similarly,  a  delay  should  not  be  misinterpreted  as  a  lessening  of  support 
for  the  Joint  Strike  Fighter  program.  Rather,  it  would  demonstrate 
decisionmakers’  willingness  to  make  the  up-front  investment  necessary  to 
mature  key  technologies  before  committing  the  Joint  Strike  Fighter  team 
to  deliver  a  product.  Such  a  commitment  is  more  likely  to  put  the  program 
on  a  better  footing  to  succeed  than  placing  the  burden  on  the  engineering 
and  manufacturing  development  phase. 


Mr.  Chairman,  that  concludes  my  statement.  I  will  be  happy  to  respond  to 
any  questions  you  or  other  Members  of  the  Subcommittee  may  have. 


Contacts  and 
Acknowledgments 


For  future  questions  regarding  this  testimony,  please  contact  Louis 
Rodrigues,  (202)512-4841.  Individuals  making  key  contributions  to  this 
testimony  include  Katherine  Schinasi,  Paul  Francis,  Michael  Sullivan, 
Matthew  Lea,  and  Katrina  Taylor. 


Page  19 


GAO/T-NSIAD-OO-1 73 


Related  GAOProducts 


Defense  Acquisition:  Employing  Best  Practices  Can  Shape  Better  Weapon 
System  Decisions  (GAO/T-NSIAD-OO-137,  Apr.  26,  2000). 

Best  Practices:  POD  Training  Can  Do  More  to  Help  Weapon  System 
Program  Implement  Best  Practices  (GAO/NSIAD-99-206,  Aug.  16,  1999). 

Best  Practices:  Better  Management  of  Technology  Development  Can 
Improve  Weapon  System  Outcomes  (GAO/NSIAD-99-162,  July  30,  1999). 

Defense  Acquisitions:  Best  Commercial  Practices  Can  Improve  Program 
Outcomes  (GAO/T-NSIAD-99-116,  Mar.  17,  1999).  _ 

Defense  Acquisition:  Improved  Program  Outcomes  Are  Possible  (GAO/T- 
NSIAD-98-123,  Mar.  17,  1998). 

Best  Practices:  POD  Can  Help  Suppliers  Contribute  More  to  Weapon 
System  Programs  (GAO/NSIAD-98-87,  Mar.  17,  1998). 

Best  Practices:  Successful  Application  to  Weapon  Acquisition  Requires 
Changes  in  POPs  Environment  (GAO/NSIAD-98-56,  Feb.  24,  1998). 

Major  Acquisitions:  Significant  Changes  Underway  in  POPs  Earned  Value 
Management  Process  (GAO/NSIAD-97-108,  May  5,  1997). 

Best  Practices:  Commercial  Quality  Assurance  Practices  Offer 
Improvements  for  POD  (GAO/NSIAD-96-162,  Aug.  26,  1996). 


(707509) 


Page  20 


GAO/T-NSIAD-OO-173 


Ordering  Information 


To  Report  Fraud, 
Waste,  and  Abuse  in 
Federal  Programs 


Contact  one: 


Web  site:  http://wynv.gao^ov/fr£iudnet/fraudnet.htm 


E-mail:  fraudnet@gao.gov  \ 
1-800-424-5424  (automated  answering  system) 


